Imaging apparatus

ABSTRACT

An imaging apparatus includes: an image sensor that captures a subject image to generate image data; a first depth measurer that acquires first depth information indicating a depth at a first spatial resolution, the depth showing a distance between the imaging apparatus and a subject in an image indicated by the image data; a second depth measurer that acquires second depth information indicating the depth in the image at a second spatial resolution different from the first spatial resolution; and a controller that acquires third depth information indicating the depth at the first or second spatial resolution for each region of different regions in the image, based on the first depth information and the second depth information.

BACKGROUND 1. Technical Field

The present disclosure relates to an imaging apparatus having a depthmeasurement function of measuring a depth to a subject.

2. Related Art

JP 2014-063142 A discloses a depth detection device used for an imagingapparatus such as a digital camera. The depth detection device includesan arithmetic unit that detects a subject depth to a subject on thebasis of a first signal (A image signal) generated by an image sensordue to a light flux passing through a first pupil region of an imagingoptical system, and a second signal (B image signal) due to a light fluxpassing through a second pupil region different from the first pupilregion of the imaging optical system. The arithmetic unit performs afirst process of calculating a subject depth and the like based on thefirst signal and the second signal by a phase detection method, and asecond process of calculating a subject depth and the like based on thefirst signal and the second signal that are the same as those in thephase detection method by a DFD method. In the image sensor of JP2014-063142 A, a photoelectric converter as an A pixel and aphotoelectric converter as a B pixel are arranged in each pixel.

SUMMARY

The present disclosure provides an imaging apparatus capable ofaccurately obtaining a depth to a subject.

An imaging apparatus according to the present disclosure includes: animage sensor that captures a subject image to generate image data; afirst depth measurer that acquires first depth information indicating adepth at a first spatial resolution, the depth showing a distancebetween the imaging apparatus and a subject in an image indicated by theimage data; a second depth measurer that acquires second depthinformation indicating the depth in the image at a second spatialresolution different from the first spatial resolution; and a controllerthat acquires third depth information indicating the depth at the firstor second spatial resolution for each region of different regions in theimage, based on the first depth information and the second depthinformation.

According to the imaging apparatus of the present disclosure, a depth tothe subject can be accurately obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a digital cameraaccording to a first embodiment of the present disclosure;

FIG. 2 is a diagram for explaining sensor pixels in an image sensor ofthe digital camera;

FIGS. 3A to 3D are a diagrams for explaining an outline of an operationof a depth measurement function of the digital camera;

FIG. 4 is a flowchart for explaining an operation of the digital camera;

FIG. 5 is a flowchart illustrating depth map combining processing in thedigital camera;

FIGS. 6A to 6E are diagrams for explaining depth map combiningprocessing in the digital camera;

FIG. 7 is a flowchart illustrating display processing of an AF frame inthe digital camera; and

FIGS. 8A and 8B are diagrams illustrating a display example of the AFframe in the digital camera.

DETAILED DESCRIPTION

In the following, an embodiment will be described in detail withreference to the drawings as appropriate. However, unnecessarilydetailed descriptions will be omitted in some cases. For example,detailed descriptions of already well-known matters and repetition ofdescriptions of substantially the same configuration will be omitted insome cases. This is to prevent the following description from beingunnecessary redundant and to facilitate those skilled in the art tounderstand the present disclosure. Note that the inventor(s) providesthe accompanying drawings and the following description in order forthose skilled in the art to fully understand the present disclosure, anddoes not intend to limit the subject matter described in the claims bythe accompanying drawings and the following description.

First Embodiment

In a first embodiment, a digital camera having a depth measurementfunction by two types of depth measurement methods will be described asan example of an imaging apparatus according to the present disclosure.

1. Configuration

A configuration of a digital camera according to the first embodimentwill be described with reference to FIGS. 1 and 2 .

FIG. 1 is a diagram illustrating a configuration of a digital camera 100according to the present embodiment. The digital camera 100 of thepresent embodiment includes an image sensor 115, an image processingengine 120, a display monitor 130, and a controller 135. Further, thedigital camera 100 includes a buffer memory 125, a card slot 140, aflash memory 145, a user interface 150, and a communication module 155.For example, the digital camera 100 further includes an optical system110 and a lens driver 112.

The optical system 110 includes a focusing lens, a zooming lens, anoptical camera-shake correction lens (OIS), a diaphragm, a shutter, andthe like. The focusing lens is a lens for changing a focusing state of asubject image formed on the image sensor 115. The zooming lens is a lensfor changing a magnification ratio of a subject image formed by theoptical system. Each of the focusing lens and the like includes one ormore lenses.

The lens driver 112 drives the focusing lens and the like in the opticalsystem 110. The lens driver 112 includes a motor, and moves the focusinglens along an optical axis of the optical system 110 under the controlof the controller 135. A configuration for driving the focusing lens inthe lens driver 112 can be realized by a DC motor, a stepping motor, aservo motor, an ultrasonic motor, or the like.

The image sensor 115 captures a subject image formed via the opticalsystem 110 to generate image-capturing data. The image-capturing dataconstitutes image data representing a captured image by the image sensor115. For example, the image sensor 115 generates image data of newframes at a predetermined frame rate (e.g., 30 frames/second). Thecontroller 135 controls a timing of generation of image-capturing databy the image sensor 115 and operation of an electronic shutter. As theimage sensor 115, it is possible to use various image sensors such as acomplementary metal-oxide semiconductor (CMOS) image sensor, acharge-coupled device (CCD) image sensor, and a negative-channel metaloxide semiconductor (NMOS) image sensor.

The image sensor 115 performs an image capturing operation of a stillimage, an image capturing operation of a through image, and the like.The through image is mostly a moving image and is displayed on thedisplay monitor 130 for a user to determine a composition for imagecapturing of a still image. The through image and the still image areeach an example of a captured image in the present embodiment. The imagesensor 115 is an example of an image sensor in the present embodiment.

The image sensor 115 of the present embodiment includes sensor pixels116 of an image plane phase detection method. FIG. 2 is a diagram forexplaining the sensor pixels 116 in the image sensor 115.

For example, as illustrated in FIG. 2 , the sensor pixels 116 of theimage plane phase detection method are arranged on an image plane of theimage sensor 115 instead of pixels for image capturing. In other words,the image sensor 115 has pixels shielded from light in image capturingby the number of the sensor pixels 116. A plurality of the sensor pixels116 are arranged at positions to be depth measurement targets,respectively, in a depth measurement function of the image plane phasedetection method on the image plane of the image sensor 115, andconstitute depth measurement points of the image plane phase detectionmethod. For example, each of the sensor pixels 116 includes aphotoelectric converter or the like divided so as to form two types ofoptical images obtained by pupil division in the optical system 110.

Returning to FIG. 1 , the image processing engine 120 performs variousprocesses on the image-capturing data output from the image sensor 115to generate image data, or performs various processes on the image datato generate an image to be displayed on the display monitor 130.Examples of the various processes include, but are not limited to,interpolation processing of light-shielded pixels corresponding to thesensor pixels 116, white balance correction, gamma correction, YCconversion processing, electronic zoom processing, compressionprocessing, expansion processing, and the like. The image processingengine 120 may be configured with a hard-wired electronic circuit, maybe configured with a microcomputer or a processor using a program, ormay be configured with other elements.

In the present embodiment, the image processing engine 120 includes adepth from defocus (DFD) calculator 121 that implements a depthmeasurement function of a DFD method, and a phase detection (PD)calculator 122 that implements a depth measurement function of the imageplane phase detection method. In the present embodiment, the DFDcalculator 121 is an example of a first depth measurer, and the phasedetection calculator 122 is an example of a second depth measurer. Eachof the calculators 121 and 122 includes an arithmetic circuit for eachdepth measurement function, for example. Each of the calculators 121 and122 will be described later.

The display monitor 130 is an example of a display that displays variousinformation. For example, the display monitor 130 displays an image(through image) represented by image data, which is captured by theimage sensor 115 and is subjected to image processing of the imageprocessing engine 120. The display monitor 130 further displays a menuscreen for a user to set various settings of the digital camera 100 orother screens. The display monitor 130 can be configured with a liquidcrystal display device or an organic electroluminescence (EL) device,for example.

The user interface 150 is a generic term for e.g. hardware keys such asoperation buttons and operation levers provided on an outer casing ofthe digital camera 100 and receives an operation by a user. For example,the user interface 150 includes a shutter release button, a mode dial,and a touch panel. When receiving a user's operation, the user interface150 transmits to the controller 135 an operation signal corresponding tothe user's operation.

The controller 135 collectively controls an overall operation of thedigital camera 100. The controller 135 includes a central processingunit (CPU) or the like, and the CPU executes a program (software) torealize predetermined functions. Instead of the CPU, the controller 135may include a processor configured with a dedicated electronic circuitdesigned to realize predetermined functions. That is, the controller 135can be realized by various processors such as a CPU, a microprocessorunit (MPU), a graphic processor unit (GPU), a digital signal processor(DSP), a field programmable gate array (FPGA), and an applicationspecific integrated circuit (ASIC). The controller 135 may be configuredwith one or a plurality of processors. The controller 135 may beconfigured with one semiconductor chip together with the imageprocessing engine 120 or other elements.

The buffer memory 125 is a recording medium that functions as a workmemory of the image processing engine 120 and the controller 135. Thebuffer memory 125 is realized by a dynamic random access memory (DRAM)or other components. The flash memory 145 is a nonvolatile recordingmedium. Further, not shown in the drawings, the controller 135 mayinclude various internal memories and may incorporate a read-only memory(ROM), for example. The ROM stores various programs to be executed bythe controller 135. Furthermore, the controller 135 may incorporate aRAM that functions as a work area of the CPU.

The card slot 140 is a means by which a detachable memory card 142 isinserted. The card slot 140 can electrically and mechanically connectthe memory card 142. The memory card 142 is an external memory includinga recording element such as a flash memory therein. The memory card 142can store data such as image data generated by the image processingengine 120.

The communication module 155 is a communication module (circuit) thatperforms communication conforming to the communication standard IEEE802.11, the Wi-Fi standard, or the like. The digital camera 100 cancommunicate with other devices via the communication module 155. Thedigital camera 100 may directly communicate with another device via thecommunication module 155, or may communicate via an access point. Thecommunication module 155 may be connectable to a communication networksuch as the Internet.

2. Operation

A description will be given below on the operation of the digital camera100 configured as described above.

The digital camera 100 according to the present embodiment generatesdepth information indicating a depth to a subject with higher accuracyby combining two depth measurement techniques having differentperformances. An outline of the operation of the depth measurementfunction of the digital camera 100 will be described with reference toFIGS. 3A to 3D.

FIG. 3A illustrates a captured image Im in the digital camera 100 of thepresent embodiment. The captured image Im constitutes one frame ofvarious moving images including a through image, for example.Hereinafter, a horizontal direction in the captured image Im is definedas an X direction, and a vertical direction is defined as a Y direction.Further, a depth direction from the digital camera 100 to the subjectmay be referred to as a Z direction.

FIGS. 3B to 3D illustrate various depth maps M1 to M3 corresponding tothe captured image Im of FIG. 3A, respectively. The depth maps M1 to M3are examples of depth information indicating a depth measurement resultfor each depth measurement point in a two-dimensional coordinate system(X, Y) similar to the captured image Im, for example. In the depth mapsM1 to M3 illustrated in FIGS. 3B to 3D, a depth value of the depthmeasurement result is shown by the shade of gray. As the gray islighter, the depth is closer. As the gray is darker, the depth isfarther.

The digital camera 100 according to the present embodiment performsdepth measurement by the DFD method, to generate a DFD depth map M1 asillustrated in FIG. 3B, for example. The DFD depth map M1 is a depth mapindicating a depth measurement result of the DFD method. Furthermore,the digital camera 100 according to the present embodiment furtherperforms depth measurement by the image plane phase detection method,and generates a phase detection depth map M2 (FIG. 3C) indicating aresult of this depth measurement.

As illustrated in FIG. 3B, the DFD depth map M1 has a relatively highspatial resolution, for example. The spatial resolution is variousresolutions for detecting a spatial position of a subject in at leastany direction in a three-dimensional space, for example. For example,the spatial resolution includes a two-dimensional resolution in the Xand Y directions (i.e., the number of depth measurement points) and aresolution in the Z direction (i.e., a depth measurement accuracy).

FIGS. 3A to 3D illustrate an example in which two persons 50 and 51 areincluded in the subject in the captured image Im, and one person 50 isfocused. In this example, the other person 51 is out of focus.

In the DFD depth map M1 of the present example, as illustrated in FIG.3B, an outer shape of the focused person 50 appears. Thus, a subjectsuch as the person 50 can be detected with high accuracy. On the otherhand, the person 51 out of focus cannot be so detected. As describedabove, it is conceivable that a depth measurement accuracy of the DFDdepth map M1 may be obtained with high accuracy only in a depth rangeclose to a focus position of the digital camera 100. That is, when thedepth is farther from the focus position, the depth measurement accuracymay decrease due to large blurring with an excessive amount of blurring.

In the phase detection depth map M2, as illustrated in FIG. 3C, it isassumed that the spatial resolution is lower than the spatial resolutionof the DFD depth map M1, for example (FIG. 3B). For example, thetwo-dimensional resolution in the phase detection depth map M2 islimited with the correspondence to the number of the sensor pixels 116provided in the image sensor 115. In addition, it is considered that adepth measurement accuracy of the phase detection depth map M2 is lowerthan the depth measurement accuracy of the DFD depth map M1 in the depthrange close to the focus position.

On the other hand, it is conceivable that the depth measurement accuracyof the phase detection depth map M2 exceeds the depth measurementaccuracy of the DFD depth map M1 outside the above-described depthrange. For example, the person 51 out of focus in the captured image Imof FIG. 3A can be detected in the phase detection depth map M2 asillustrated in FIG. 3C. As described above, the phase detection depthmap M2 is not limited to a specific depth range, and is considered tohave an advantage of a resolution that enables a stable depthmeasurement accuracy.

Therefore, the digital camera 100 of the present embodiment combines thetwo depth maps M1 and M2 having different performances as describedabove to generate a combined depth map M3 having the advantage of theDFD depth map M1 and the advantage of the phase detection depth map M2(FIG. 3D).

FIG. 3D illustrates a combined depth map M3 obtained from the DFD depthmap M1 of FIG. 3B and the phase detection depth map M2 of FIG. 3C.According to the combined depth map M3 of the present embodiment, asillustrated in FIG. 3D, an outer shape of the person 50 in focus can beobtained, and the person 51 out of focus can also be detected, forexample.

As described above, the digital camera 100 according to the presentembodiment can obtain the combined depth map M3 having the high spatialresolution by the DFD depth map M1 and the stable depth measurementresolution by the phase detection depth map M2. Hereinafter, details ofthe operation of the digital camera 100 in the present embodiment willbe described.

2-1. Overall Operation

An overall operation of the digital camera 100 according to the presentembodiment will be described with reference to FIG. 4 . FIG. 4 is aflowchart for explaining the operation of the digital camera 100.

Processing illustrated in the flowchart of FIG. 4 is performed for eachpredetermined period such as a frame period with the image sensor 115executing an image capturing operation of various moving images such asa through image. The processing of this flow is executed by thecontroller 135 of the digital camera 100, for example.

At first, the controller 135 of the digital camera 100 acquires the DFDdepth map M1 from the DFD calculator 121 (S1). The DFD depth map M1 isan example of first depth information having a first spatial resolution.

In step S1, the DFD calculator 121 performs depth measurement by the DFDmethod to generate the DFD depth map M1, by performing a DFD calculationfor each preset depth measurement point on the basis of image data suchas captured images of two frames, for example. The DFD calculation is acalculation for deriving a defocus amount (or a subject depth) on thebasis of a difference in a blur amount between frames, by e.g.calculating a point spread function, an optical transfer function or thelike. A known technique can be appropriately applied to the depthmeasurement of the DFD method (e.g., JP 2014-063142 A).

Further, the controller 135 acquires the phase detection depth map M2from the phase detection calculator 122 (S2). The phase detection depthmap M2 is an example of second depth information having a second spatialresolution. Note that a processing order of steps S1 and S2 is notparticularly limited.

In step S2, the phase detection calculator 122 performs depthmeasurement of the image plane phase detection method on the basis ofsensor signals input from the sensor pixels 116 in the image sensor 115,to generate the phase detection depth map M2. The depth measurement bythe image plane phase detection method can be performed by calculating adefocus amount or the like according to a difference between two typesof optical images by pupil division from the sensor signals for eachdepth measurement point by the sensor pixels 116, for example. A knowntechnique can be appropriately applied to the depth measurement of theimage plane phase detection method (e.g., JP 2014-063142 A).

Next, the controller 135 of the present embodiment combines the DFDdepth map M1 and the phase detection depth map M2, based on the acquireddepth maps M1 and M2 (S3). The combined depth map M3 generated in depthmap combining processing (S3) is an example of third depth informationin the present embodiment.

The depth map combining processing (S3) of the present embodimentgenerates the combined depth map M3 by comparing data of the depth mapsM1 and M2 with each other for each individual region in the capturedimage Im to adopt more accurate depth data thereto. Details of theprocessing in step S3 will be described later.

Next, the controller 135 performs various controls in the digital camera100 using the generated combined depth map M3 (S4). For example, thecontroller 135 controls an autofocus (AF) operation or displaysinformation such as an AF frame indicating a detection result of asubject to be subjected to the AF operation, based on the combined depthmap M3.

After the control using the combined depth map M3 (S4), the controller135 ends the processing illustrated in this flowchart.

According to the operation of the digital camera 100 described above,from the DFD depth map M1 and the phase detection depth map M2 (S1, S2),the combined depth map M3 with improved spatial resolution and depthmeasurement resolution can be acquired (S3). Such a highly accuratecombined depth map M3 can be used for various controls and the like inthe digital camera 100 (S4).

For example, according to a high spatial resolution in the combineddepth map M3, in the example of FIGS. 3A to 3D, the digital camera 100can extract a region along a shape of the person 50. For example, theregion extracted in this manner can used for calculation of an AFevaluation value of a contrast AF method or the like, so that it ispossible to realize high-accuracy AF control wherein mis-focusing onbackground and the like are suppressed. Further, by displaying theextracted shape of the person 50 on the display monitor 130 or the like,it is also possible to visualize the high detection accuracy of thedigital camera 100 for a user. An example of such visualizationprocessing will be described later.

Furthermore, the digital camera 100 uses the combined depth map M3 forthe AF control, so that even a subject appearing small on the capturedimage Im can be accurately focused, based on the depth data of the highspatial resolution by the DFD depth map M1. Also in the case ofre-focusing on the subject away from the focus position, such as theperson 51 in FIG. 3A, the AF control can work at high speed over a widerange of the depth measurement resolution by the phase detection depthmap M2 in the combined depth map M3.

2-2. Depth Map Combining Processing

The depth map combining processing in step S3 of FIG. 4 will bedescribed with reference to FIGS. 5 and 6 .

FIG. 5 is a flowchart illustrating the depth map combining processing inthe digital camera 100. The processing illustrated in this flow isexecuted by the controller 135 controlling the image processing engine120, for example.

FIGS. 6A and 6B illustrate a DFD depth map M1 and a phase detectiondepth map M2 before combining, respectively. The process illustrated inFIG. 5 is started in a state where the depth maps M1 and M2 are acquiredin steps S1 and S2 in FIG. 4 as illustrated in FIGS. 6A and 6B, forexample.

At first, the controller 135 performs processing to normalize a regionin the low-resolution phase detection depth map M2 so as to match thehigh-resolution DFD depth map M1, for example (S11). The processing instep S11 will be described with reference to FIGS. 6A to 6C.

FIGS. 6A and 6B illustrate data points P1 and P2 corresponding to theresolutions of the depth maps M1 and M2, respectively. Each of the datapoints P1 and P2 has a depth value as a result of depth measurementperformed for each depth measurement point at a corresponding positionin the captured image Im.

As illustrated in FIG. 6A, the DFD depth map M1 includes a data regionhaving a size including a large number of data points P1 for a highresolution. On the other hand, as illustrated in FIG. 6B, the phasedetection depth map M2 includes a data region including N data points P2smaller than the number of data points P1 of the DFD depth map M1.Therefore, in step S11, the data region of the phase detection depth mapM2 is normalized in accordance with the DFD depth map M1.

FIG. 6C illustrates a normalized phase detection depth map M2′. Thenormalized phase detection depth map M2′ is configured by dividing adata region having the same size as the DFD depth map M1 into N dividedregions R2-1 to R2-N. For example, in step S11, the controller 135manages the normalized phase detection depth map M2′ by allocating thedepth value of each data point P2 in the phase detection depth map M2acquired in step S2 as depth data for each of the divided regions R2-1to R2-N.

Next, the controller 135 selects one divided region R2 in order from thefirst to N-th divided regions R2-1 to R2-N in the normalized phasedetection depth map M2′ (S12). In steps S12 to S18, processing tocompare the DFD depth map M1 and the phase detection depth map M2′ issequentially performed for each selected one of divided regions R2.

The controller 135 extracts region R1 corresponding to currentlyselected divided region R2 and its depth data from the DFD depth map M1(S13). FIG. 6D illustrates corresponding regions R1-1 to R1-N of thefirst to N-th divided regions R2-1 to R2-N of FIG. 6C in the DFD depthmap M1 of FIG. 6A. Each corresponding region R1 of the DFD depth map M1includes a plurality of data points P1 as a depth measurement result ofthe same region as the divided region R2 of the corresponding phasedetection depth map M2′ on the captured image Im.

Based on the depth data in the corresponding region R1 extracted fromthe DFD depth map M1, the controller 135 calculates an evaluation valueof the DFD depth measurement for the currently selected divided regionR2, for example (S14). For example, the evaluation value of the DFDdepth measurement is an average value over the depth value of each datapoint P1 in the corresponding region R1 of the DFD depth map M1, and iscalculated by an arithmetic mean, a geometric mean, or various weightedmeans, for example.

Next, as a comparison between the two depth maps M1 and M2′ in thecurrently selected divided region R2, the controller 135 determineswhether or not the calculated evaluation value of the DFD depthmeasurement is within a predetermined allowable range according to thedepth data of the phase detection depth map M2′, for example (S15). Forexample, the allowable range is set to a depth range of an allowableerror in which the depth value allocated to the divided region R2 beingselected in the phase detection depth map M2′ and the evaluation valueof the DFD depth measurement are presumed to match.

When determining that the evaluation value of the DFD depth measurementfor the currently selected divided region R2 falls within the allowablerange (YES in S15), the controller 135 determines to adopt the depthdata of the corresponding region R1 of the DFD depth map M1 as thecombined depth map M3 (S16). In this case, the DFD depth map M1 is notparticularly deviated from the phase detection depth map M2′, and thedepth measurement result of the corresponding region R1 is considered tobe highly accurate.

On the other hand, when determining that the evaluation value of the DFDdepth measurement for the currently selected divided region R2 is notwithin the allowable range (NO in S15), the controller 135 determines toadopt the depth data of the phase detection depth map M2′ as thecombined depth map M3 (S17). In this case, as the DFD depth map M1 isrelatively shifted from the phase detection depth map M2′, it ispossible to detect that the depth measurement accuracy has deteriorationdue to large blurring or the like in the corresponding region R2.

When the depth data to be adopted is not determined in all the dividedregions R2-1 to R2-N (NO in S18), the controller 135 selects a newdivided region R2 to perform the processing in and after step S12. As aresult, the comparison (S12 to S18) between the two depth maps M1 andM2′ is repeated for each divided region R2.

When the depth data to be adopted is determined in all the dividedregions R2-1 to R2-N(YES in S18), the controller 135 generates thecombined depth map M3 as a combined result of the two depth maps M1 andM2 (S19). FIG. 6E illustrates the combined depth map M3 in step S19.

FIG. 6E illustrates the combined depth map M3 generated from the depthmaps M1 and M2′ of FIGS. 6C and 6D. The combined depth map M3 includes aDFD region R31 which is a region having depth data of the DFD depth mapM1 and a phase detection region R32 which is a region having depth dataof the phase detection depth map M2′. For example, the combined depthmap M3 is configured by dividing the same data region as the DFD depthmap M1 into N divided regions similarly to the corresponding regionsR1-1 to R1-N.

For example, in step S19, the controller 135 allocates the depth data ofthe corresponding region R1 determined in step S16 to the divided regionin the DFD region R31 in the combined depth map M3. The controller 135allocates the depth data of the divided region R2 determined in step S17to the divided region in the phase detection region R32 in the combineddepth map M3. In this manner, the DFD depth map M1 and the phasedetection depth map M2′ can be combined to generate the combined depthmap M3 (S19).

When generating the combined depth map M3 as described above (S19), thecontroller 135 ends the depth map combining processing (S3 in FIG. 4 )and proceeds to the processing of step S4, for example.

According to the depth map combining processing (S3) described above, itis possible to obtain the combined depth map M3 including the DFD regionR31 having a higher spatial resolution by the DFD method, and the phasedetection region R32 having stable depth measurement accuracy in a widerdepth range by the image plane phase detection method (S19).

In addition, the processing of comparing the depth maps M1 and M2 foreach of the divided regions R2 of the normalized phase detection depthmap M2′ and adopting one of the depth data can make it easy to generatethe combined depth map M3 (S11 to S19). For example, by comparing theevaluation value of the DFD depth measurement with the depth value ofthe phase detection depth map M2′ for each divided region R2 (S15), itis possible to easily determine whether or not the depth measurementaccuracy is deteriorated in the corresponding region R1 of the DFD depthmap M1.

The evaluation value of the DFD depth measurement in steps S14 and S15is not particularly limited to the average value of the depth values inthe corresponding region R1, and may be e.g. a mode value, a dispersiondegree, or a difference between the maximum value and the minimum value.In addition, the comparison target of the evaluation values of the DFDdepth measurement may not be the depth value of the phase detectiondepth map M2′. For example, reference value for determining a decreasein depth measurement accuracy may be set in advance according to thetype of the evaluation value. Furthermore, the digital camera 100 mayobtain a depth range which is before and after the vicinity of the focusposition wherein it is presumed that the depth measurement accuracy ofthe DFD method can be obtained with high accuracy, and determine whetheror not the depth data of the DFD depth map M1 is within the obtaineddepth range in step S15 or the like.

In the above description, any one of the two depth maps M1 and M2 isadopted as the combined depth map M3 for each divided region R2 (S16,S17). The digital camera 100 according to the present embodiment is notlimited to this, and for example, a portion that adopts the depth dataof the DFD depth map M1 and a portion that adopts the depth data of thephase detection depth map M2′ may be provided inside the divided regionR2.

For example, the controller 135 may determine the presence or absence ofa decrease in depth measurement accuracy for each data point P1 in thecorresponding region R1 of the DFD depth map M1, and selectively replacethe depth data of the data point P1 determined to have a decrease indepth measurement accuracy with the depth data of the phase detectiondepth map M2′. Alternatively, the digital camera 100 may generate thecombined depth map M3 by combining the two depth maps M1 and M2 so as tointerpolate the depth data of the data point P2 in the phase detectiondepth map M2 with the depth data of the data point P1 in the DFD depthmap M1.

2-3. Display Processing of AF Frame

An example of processing to visualize, to the user, that the digitalcamera 100 recognizes the shape of the subject using the combined depthmap M3 as described above will be described with reference to FIGS. 7and 8 . Hereinafter, a processing example using the combined depth mapM3 for displaying the AF frame will be described.

FIG. 7 is a flowchart illustrating display processing of the AF frame inthe digital camera 100. FIGS. 8A and 8B are diagrams illustrating adisplay example of the AF frame in the digital camera 100. Theprocessing illustrated in the flowchart of FIG. 7 is started in responseto an input of a predetermined user operation with the combined depthmap M3 generated in the digital camera 100 being stored in the buffermemory 125 or the like, for example.

At first, the controller 135 acquires position information designated bya user operation via the user interface 150 of the digital camera 100(S31). The user operation is various operations for designating aposition of a subject desired by the user on the captured image Im suchas a live view, and is, for example, a touch operation on a touch panel.

Next, the controller 135 refers to the combined depth map M3corresponding to the current captured image Im (S32), and determineswhether or not the designated position indicated by the acquiredposition information is within the DFD region R31 (FIG. 6E) in thecombined depth map M3, for example (S33).

When the designated position of the user operation is within the DFDregion R31 of the combined depth map M3 (YES in S33), the controller 135extracts the region of the subject including the designated position andthe shape thereof, based on the depth data in the combined depth map M3(S34). For example, the controller 135 performs edge analysis or thelike on the depth data of the combined depth map M3 to extract a regionalong the contour shape of the subject including the designatedposition. According to the DFD region R31 of the combined depth map M3,the shape of the designated subject can be accurately extracted from thehigh-resolution depth data.

Next, based on the information extracted from the DFD region R1 of thecombined depth map M3, the controller 135 controls the display monitor130 to display the AF frame F1 having the shape of the extracted subjectregion, for example (S35). FIG. 8A illustrates a display example of theAF frame F1 in step S35. The AF frame F1 is an example of firstdetection information.

The example of FIG. 8A illustrates a display example in a case wheresubjects of the digital camera 100 are a person 50 close to a focusposition and a person 51 far from the focus position and a useroperation for designating a position of the person 50 is input. In thepresent example, according to the user operation (S31), the controller135 of the digital camera 100 extracts the shape of the person 50 fromthe combined depth map M3 (S34). As a result, the AF frame F1 along theshape of the person 50 is displayed on the display monitor 130 (S35).

Returning to FIG. 7 , when the designated position of the user operationis not in the DFD region R31 of the combined depth map M3 (NO in S33),the designated position is in the phase detection region R32. In thiscase, the controller 135 refers to a result of image recognitionseparately performed on the captured image Im (S36), and acquires arectangular region or the like including the designated position, forexample. The image recognition in step S36 may be various processes torecognize a position of a part or the whole of a person or varioussubjects such as an animal, for example.

Next, the controller 135 controls the display monitor 130 to display arectangular AF frame F2, based on the acquired rectangular region of theimage recognition result or the like, for example (S37). FIG. 8Billustrates a display example of the AF frame F2 in step S37. The AFframe F2 is an example of second detection information.

FIG. 8B illustrates a case where the position of the person 51 far fromthe focus position is designated by the user operation from the persons50 and 51 of the same subject as in FIG. 8A. At this time, a displayformat such as a line type of the AF frame F2 may be different from thedisplay format of the AF frame F1 along the shape of the subject.Furthermore, in steps S36 and S37, the controller 135 may arrange therectangular AF frame F2 by recognizing the position, size, and the likeof the subject, based on the depth data of the combined depth map M3.

After the controller 135 causes the AF frames F1 and F2 to be displayed(S35, S37), the processing illustrated in this flowchart ends. At thistime, the display of the AF frames F1 and F2 may be updated as needed.For example, the controller 135 may repeat the processing illustrated inthis flow at a predetermined cycle. In this case, as the specifiedposition (S31) of the user operation, the position input at the time ofexecution of the first processing may be used repeatedly. Alternatively,the position of the result of tracking the subject from the firstposition may be updated. Such tracking of the subject may be performedusing the combined depth map M3 obtained sequentially, or may beperformed by image recognition as appropriate.

According to the above AF frame display processing, the digital camera100 can display the AF frame F1 along the shape of the subject for thesubject having depth measurable with higher accuracy by the DFD regionR31 of the combined depth map M3 (see FIG. 8A). As a result, it ispossible to visualize the user that the digital camera 100 recognizesthe shape of the subject with high accuracy.

Furthermore, even in a case where the subject designated by the useroperation is outside the DFD region R31 (see FIG. 8B), the digitalcamera 100 can perform focusing on the subject at high speed using thecombined depth map M3. Then, in the combined depth map M3 obtainedthereafter, the designated subject is expected to be located in the DFDregion R1. Therefore, even when displaying the rectangular AF frame F2on the person 51 initially designated as illustrated in FIG. 8B inresponse to the input of the user operation, the digital camera 100 mayupdate the shape to be displayed as the AF frame F2 once the shape ofthe person 51 is extracted by the new combined depth map M3.

In the above steps S33 to S35, an example in which the AF frame F1 alongthe shape of the subject is displayed when the designated position is inthe DFD region R31 in the combined depth map M3 (YES in S33) has beendescribed. When extracting the shape of the subject from the combineddepth map M3 (S34), the controller 135 may determine whether or not thecontour shape falls within the DFD region R31.

In a case where the shape of the subject extracted as described abovedoes not fit in the DFD region R31, the resolution of the phasedetection region R32 is partially included, resulting in reducingrecognition accuracy of the shape. In such a case, the controller 135may display a rectangular AF frame F2 instead of the AF frame F1 alongthe extracted shape.

Alternatively, the controller 135 may display the AF frame F1 along theextracted shape including the phase detection region R32 as describedabove, with the line type and the like being changed from the othercases. In this manner, the degree of accuracy of recognition of theshape by the digital camera 100 may be visualized to the user.Furthermore, the shape of the AF frame F1 may not completely match theshape extracted from the combined depth map M3, and interpolation,smoothing, or the like may be appropriately performed.

3. Summary

As described above, the digital camera 100 as an example of the imagingapparatus according to the present embodiment includes the image sensor115 as an example of the image sensor, the DFD calculator 121 as anexample of the first depth measurer, the phase detection calculator 122as an example of the second depth measurer, and the controller 135. Theimage sensor 115 captures a subject image to generate image data. TheDFD calculator 121 acquires the DFD depth map M1 which is an example offirst depth information indicating, at a first spatial resolution, adepth showing a distance between the imaging apparatus and the subjectin the image indicated by the image data (S1). The phase detectioncalculator 122 acquires the phase detection depth map M2 which is anexample of second depth information indicating a depth in an image at asecond spatial resolution different from the first spatial resolution(S2). On the basis of the DFD depth map M1 and the phase detection depthmap M2, the controller 135 acquires the combined depth map M3 which isan example of third depth information indicating a depth in the first orsecond spatial resolution for each region of different regions in theimage (S3).

According to the digital camera 100 described above, the depth to thesubject can be accurately obtained by the combined depth map M3 based onthe DFD depth map M1 and the phase detection depth map M2 havingdifferent spatial resolutions. According to the combined depth map M3,as the spatial resolution of the DFD depth map M1 and the spatialresolution of the phase detection depth map M2 are included for eachregion on the image, an accurate depth measurement result can beobtained.

In the present embodiment, the combined depth map M3 includes depthinformation with higher depth measurement accuracy out of the DFD depthmap M1 and the phase detection depth map M2 for each region in theimage. As a result, highly accurate depth measurement accuracy can beobtained in the combined depth map M3.

In the present embodiment, the depth measurement accuracy of the DFDdepth map M1 is higher than that of the phase detection depth map M2within a specific depth range. The depth measurement accuracy of thephase detection depth map M2 is higher than that of the DFD depth map M1outside the depth range. The combined depth map M3 includes depth dataof the DFD depth map M1 within the depth range and depth data of thephase detection depth map M2 outside the depth range. As a result, thedepth data in which the DFD depth map M1 and the phase detection depthmap M2 have higher depth measurement accuracy inside and outside theabove-described depth range is included in the combined depth map M3,and the depth measurement accuracy of the combined depth map M3 can beimproved.

In the present embodiment, the specific depth range is a range includinga focus position focused on the image sensor 115. The DFD calculator 121generates the DFD depth map M1 on the basis of the image data generatedby the image sensor 115. In the depth range near the focus position inthe digital camera 100, the depth measurement accuracy of the combineddepth map M3 can be improved using the depth data of the DFD depth mapM1.

In the present embodiment, the spatial resolution (first spatialresolution) of the DFD depth map M1 is higher than the spatialresolution (second spatial resolution) of the phase detection depth mapM2 in the two-dimensional direction (X, Y) corresponding to the image.As a result, in the region using the depth data of the DFD depth map M1,high resolution is obtained in the combined depth map M3. On the otherhand, the spatial resolution of the phase detection depth map M2 mayinclude a more stable depth measurement resolution than the DFD depthmap M1 in the Z direction of the depth.

In the present embodiment, the DFD calculator 121 performs depthmeasurement by a depth from defocus (DFD) method to generate the DFDdepth map M1. The phase detection calculator 122 performs depthmeasurement by the image plane phase detection method to generate thephase detection depth map M2. The depth maps M1 and M2 based on suchdifferent depth measurement methods can be combined to obtain theaccurate combined depth map M3.

In the present embodiment, the controller 135 controls the focusingoperation to focus on the subject, based on the combined depth map M3(S4). According to the combined depth map M3, it is possible to easilycontrol the focusing operation by the depth measurement result with highaccuracy.

In the present embodiment, the digital camera 100 further includes thedisplay monitor 130 that is an example of a display that displays animage indicated by image data. The controller 135 causes the AF frameF1, which is an example of first detection information having a shapealong the subject in the image, to be displayed on the display monitor130 on the basis of the combined depth map M3 (S35). A highly accuratedepth measurement result by the combined depth map M3 can be visualizedto the user by the AF frame F1 or the like along the subject, and thedigital camera 100 can be easily used.

In the present embodiment, in a case where the subject is located in theDFD region R1 as an example of a region having a first spatialresolution in the combined depth map M3 (YES in S33), the controller 135displays the AF frame F1 of the first detection information (S35). In acase where the subject is not located in the DFD region R1 in thecombined depth map M3 (NO in S33), the controller 135 displays the AFframe F2 as an example of second detection information having a shapedifferent from the first detection information (S37). As a result, whenthe digital camera 100 can accurately recognize the shape of thesubject, the first detection information can be selectively visualized,and the user can easily use the digital camera 100.

In the present embodiment, the digital camera 100 includes the imagesensor 115 that captures a subject image and generates image data, thedisplay monitor 130 that displays an image indicated by the image data,and the controller 135 that controls the display monitor 130 on thebasis of depth information indicating a depth between the subject in theimage indicated by the image data and the imaging apparatus. In a casewhere the subject is located within a specific depth range (YES in S33),the controller 135 may cause the display monitor 130 to display firstdetection information having a shape along the subject in the image(S35). As a result, in the digital camera 100, when the shape of thesubject can be recognized by accurately obtaining the depth to thesubject, such a recognition result can be visualized for the user, andthe digital camera 100 can be easily used.

In the present embodiment, when the subject is not located within thedepth range (NO in S33), the controller 135 may cause the displaymonitor 130 to display the second detection information having a shapedifferent from the first detection information (S37). As a result, evenwhen the depth to the subject cannot be obtained accurately or the shapeof the subject cannot be recognized in the digital camera 100, the usercan easily use the digital camera 100 by the display of the displaymonitor 130.

OTHER EMBODIMENTS

As described above, the first embodiment has been described as anexample of the technique disclosed in the present application. However,the technique in the present disclosure is not limited thereto, and canalso be applied to embodiments in which changes, replacements,additions, omissions, and the like are made as appropriate. Further, anew embodiment can be made by combining the components described in thefirst embodiment.

In the first embodiment described above, the DFD calculator 121 and thephase detection calculator 122 have been exemplified as examples of thefirst and second depth measurers. In the present embodiment, the firstand second depth measurers are not limited thereto, and variousconfigurations using various depth measurement methods can be applied.For example, a time of flight (TOF) method, a range finder, a binocularstereo depth measurement method, a color-discriminated depth measurementmethod, depth estimation by artificial intelligence such as machinelearning, or the like may be applied to each depth measurer. That is, inthe present embodiment, the depth information may be acquired by depthmeasurement of various active sensing methods. Further, an evaluationvalue of a contrast AF method or the like may be used as the depthinformation of the present embodiment.

In the above embodiments, an example in which the combined depth map M3is used for controlling the digital camera 100 has been described. Theapplication of the combined depth map M3 is not limited thereto, and thecombined depth map M3 may be used for image processing of variouscaptured images at the time of generating the combined depth map M3, forexample. In this case, the combined depth map M3 may be output from thedigital camera 100 to the outside together with the image data of thecaptured image, and may be used for post-processing such as editing ofimage data including a moving image, for example. Further, the combineddepth map M3 may be used for scene recognition or various determinationprocessing.

In the above embodiments, the various depth maps M1 to M3 areexemplified as an example of the first to third depth information. Inthe present embodiment, the first to third depth information is notlimited to the depth map, and may be various information indicating ameasurement result of the depth to the subject, such as athree-dimensional point group or a depth image. Furthermore, the variousdepth information may be represented by various amounts corresponding tothe depth to the subject, and may be represented by a defocus amount,for example.

In the above embodiments, the AF frame F1 indicating the shape of thesubject is exemplified as the first detection information by using thecombined depth map M3 obtained by combining the two depth maps M1 andM2. In the present embodiment, based on not necessarily limited to thecombined distance map M3 but the various depth information indicating adepth measurement result with high accuracy to such an extent that theshape of the subject can be extracted, the digital camera 100 maygenerate the first detection information and display the first detectioninformation on the display.

In the above embodiments, an example has been described in which the AFframe is displayed in response to the input of the designated positionby the user operation in the AF frame display processing (FIG. 7 ). Inthe present embodiment, the digital camera 100 may display the AF framewithout being limited to the input of the user operation, and in thiscase as well, the shape of the AF frame can be set with reference to thecombined depth map M3 as described above. Furthermore, the digitalcamera 100 may use the combined depth map M3 in the same manner asdescribed above when displaying a display frame for a plurality ofsubjects, such as a candidate frame of an AF target, in addition to theAF frame for the subject of the AF target. For example, the digitalcamera 100 may determine whether or not the position where the subjectis detected by the image recognition of the captured image Im is the DFDregion R31 and set the shape of the display frame. Such a display frameis also an example of the first or second detection informationaccording to the shape thereof.

In the above embodiments, the digital camera 100 including the opticalsystem 110 and the lens driver 112 has been exemplified. The imagingapparatus of the present embodiment may not include the optical system110 and the lens driver 112, and may be an interchangeable lens typecamera, for example.

In the above embodiments, a digital camera is described as an example ofthe imaging apparatus, but the imaging apparatus is not limited to thedigital camera. The imaging apparatus of the present disclosure only hasto be electronic equipment having an image capturing function (e.g., avideo camera, a smartphone, and a tablet terminal).

In the above, the embodiments are described as examples of thetechniques in the present disclosure. For that purpose, the accompanyingdrawings and the detailed description are provided.

Therefore, the components illustrated in the accompanying drawings anddescribed in the detailed description not only include componentsessential for solving the problem but also can include, to exemplify thetechniques, components that are not essential for solving the problem.For this reason, it should not be immediately recognized that thoseunnecessary components are necessary only because those unnecessarycomponents are described in the accompanying drawings or the detaileddescription.

In addition, since the above-described embodiments are intended toillustrate the technique in the present disclosure, various changes,replacements, additions, omissions, and the like can be made within thescope of the claims or equivalents thereof.

The present disclosure is applicable to various imaging apparatuseshaving a depth measurement function.

1. An imaging apparatus comprising: an image sensor that captures asubject image to generate image data; a first depth measurer thatacquires first depth information indicating a depth at a first spatialresolution, the depth showing a distance between the imaging apparatusand a subject in an image indicated by the image data; a second depthmeasurer that acquires second depth information indicating the depth inthe image at a second spatial resolution different from the firstspatial resolution; and a controller that acquires third depthinformation indicating the depth at the first or second spatialresolution for each region of different regions in the image, based onthe first depth information and the second depth information.
 2. Theimaging apparatus according to claim 1, wherein the third depthinformation includes, for each region in the image, depth informationhaving higher depth measurement accuracy among the first and seconddepth information.
 3. The imaging apparatus according to claim 1,wherein a depth measurement accuracy of the first depth information ishigher than a depth measurement accuracy of the second depth informationwithin a specific depth range, and outside the depth range, the depthmeasurement accuracy of the second depth information is higher than thedepth measurement accuracy of the first depth information, wherein thethird depth information includes the first depth information within thedepth range and the second depth information outside the depth range. 4.The imaging apparatus according to claim 3, wherein the specific depthrange is a range including a focus position focused on the image sensor.5. The imaging apparatus according to claim 1, wherein the first spatialresolution is higher than the second spatial resolution in atwo-dimensional direction corresponding to the image.
 6. The imagingapparatus according to claim 1, wherein the controller controls afocusing operation to focus on the subject, based on the third depthinformation.
 7. The imaging apparatus according to claim 1, furthercomprising a display that displays the image indicated by the imagedata, wherein the controller causes the display to display firstdetection information having a shape along the subject in the image,based on the third depth information.
 8. The imaging apparatus accordingto claim 7, wherein the controller causes the first detectioninformation to be displayed, in a case where the subject is located in aregion of the first spatial resolution in the third depth information,and causes second detection information having a shape different fromthe first detection information to be displayed, in a case where thesubject is not located in the region of the first spatial resolution inthe third depth information.
 9. A non-transitory computer-readablerecording medium storing a program for causing an electronic device tooperate as the imaging apparatus according to claim 1.